Skip to content

docs(agents): require an oracle, not a model of one, for deciding predicates - #300

Open
vsits-proxy-builder[bot] wants to merge 2 commits into
mainfrom
docs/predicate-oracle-rule
Open

docs(agents): require an oracle, not a model of one, for deciding predicates#300
vsits-proxy-builder[bot] wants to merge 2 commits into
mainfrom
docs/predicate-oracle-rule

Conversation

@vsits-proxy-builder

Copy link
Copy Markdown
Contributor

What this fixes

We approved a broken trust guard. PR #283's ca-trust check merged as 23346ac9 with two Codex rounds, independent verification of every blocker, and a green suite — and it is wrong in both directions on main today:

  • False rejectnew X509Certificate(block) runs on every PEM block and the throw escapes to the outer catch. One CRL in a bundle voids the whole file. Rejection is not the safe direction here: the fallback drops every sibling and corporate CA for the session, which is the exact failure the contract exists to prevent.
  • False acceptX509Certificate ignores the PEM label, so our CA relabelled TRUSTED CERTIFICATE yields byte-identical DER and passes carriesUs, while node's loader skips any block not labelled exactly CERTIFICATE.

Both reproduce on origin/main. Both were reproduced independently by the AI Team Lead before this rule was agreed.

The review process worked on everything it was pointed at — a write→rename race measured at 0.88 ms over 5,000 iterations, the rendezvous path grepped, file modes checked. It was pointed at the wrong thing. Nobody ever fed the guard a realistic bundle, on the function that decides what the client trusts for the host its API keys travel to.

The corollaries are the point

The headline rule alone would not have caught this, which is why both corollaries are in the section rather than as asides:

  • A. The pre-fix(launcher): match the ca-trust guard to what node's CA loader accepts #296 test did run a genuine TLS handshake — through tls.connect({ca}), while the launcher uses NODE_EXTRA_CA_CERTS. The two disagree. Green test, certified nothing.
  • B. Mutating the shipped guard to accept unconditionally left test/proxy-forward-ca.test.mjs passing 12/12, because the test exercised a hand-copied twin. "We have tests" was counted as reassurance by two reviewers.

Scope, per AITL

The rule fires on any predicate whose job is to predict another program's behavior, not only on trust decisions. The unifying property is that the oracle exists and we chose to model it instead of calling it. git push --dry-run as a test of a branch ruleset is the same failure — it reports success against a ruleset the server never consulted; we hit that one this week too.

Not in scope here

The design question — why a predicate rather than an oracle at all? — is open with Chris and @codeslake. AITL measured a 38 ms median floor for a bare node spawn on this host, so the honest version is that an oracle is not free. That decision also touches two other language implementations of the same contract and isn't made in this PR.

Non-Functional Requirements

  • Size/complexity budget — +69 lines, documentation only, one file.
  • Threat model — n/a. No code, no inputs, no execution path.
  • Maintainability constraints — sits between ## Evidence Class and ## Anti-Bloat Lens, the two sections it composes with. No new abstraction.
  • Performance/reliability — n/a.
  • Load-bearing?No in the code sense; it ships no behavior. It is binding on every reviewer, so it wants Chris's read on the wording rather than on the risk.

Ref #293

— Proxy Builder

…dicates

PR #283's ca-trust guard merged with two approvals and independent
verification of every blocker, and is wrong in both directions on main —
a CRL in a bundle voids the whole file, and our CA relabelled TRUSTED
CERTIFICATE passes a check node's loader would fail.

Neither reviewer ever fed the guard a realistic bundle. The suite was
green because it exercised a hand-copied twin: mutating the shipped
guard to accept unconditionally left it at 12/12.

Adds the rule and its two corollaries — the oracle must be the same API
production calls, and a test must be proven to reach the shipped code
before it counts as evidence — plus the phrasing rule for universal
claims like "conservative, never permissive".

Widened past trust decisions to any predicate that predicts another
program, per AI Team Lead: git push --dry-run against a branch ruleset
is the same failure with a different oracle.
@codeslake

codeslake commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Reproduced both defects on main, and measured the cost figure the "not in scope" section turns on. The rule is right; one number under it is not, and it is the number that makes the design question look closed when it isn't.

Both claims reproduce

Node v24.11.1. main's inline guard, against the loader it is predicting:

bundle main guard NODE_EXTRA_CA_CERTS loads
our CA relabelled TRUSTED CERTIFICATE ok 0
valid X509 CRL + our CA bundle does not carry our CA 1

False accept and false reject, as written.

The 38 ms is the spawn floor, not the oracle

The scope note reads "an oracle is not free" off a 38 ms median for a bare node spawn. That is the floor a spawn costs at all — it is not what asking the oracle costs on top of what this path already does. Measured on the same host, 100 interleaved runs so both share load conditions:

bare node spawn        median 16.4 ms   p95 17.9
oracle (real loader)   median 20.4 ms   p95 22.2
                       ─────────────────────────
oracle increment        median  4.0 ms

Once per launcher start, not per request. And the launcher already forks node at bin/claude-via-proxy.mjs:144 for the proxy, so a spawn is already in this path.

What the oracle buys, against ground truth

Ground truth here is not a second opinion — it is handing the bundle to a child via NODE_EXTRA_CA_CERTS and doing a real HTTPS handshake against a local server holding the leaf. Two harness bugs cost me two wrong runs first: a same-process server never accepts, because spawnSync blocks the event loop that would have to serve it; and the leaf's SAN is api.anthropic.com, so servername: "localhost" hangs rather than failing.

Against the real 132-cert bundle on this host (publishers: our CA, a third-party pin CA, system roots):

bundle handshake predicate oracle
real 132-cert OK true true
overlapping BEGIN markers FAIL true false
non-cert body containing a -----BEGIN line OK false true

Predicate 2/3 wrong; oracle 3/3 right. Those two shapes are the open blockers on #296, which is round 6 of the five this PR counts.

The generic argument, and a limit on my own evidence

The loader does not know who published a certificate. An oracle reads what it actually loaded and looks for the CA you asked about, so the same code path answers for any publisher with a different needle. A predicate parses file syntax, so every additional publisher is more shapes it has not seen — a third publisher arriving on this host is that risk increasing, not a hypothetical.

Limit, stated because the rule this PR adds is about exactly this: my harness serves our own leaf, so the rows I ran for the third-party CA prove nothing about its trust path. The handshake passed on our CA inside the bundle. The publisher-agnostic claim above is structural plus matching verdicts, not an end-to-end handshake for that publisher.

One more instance of the same class, upstream of the guard

The bundle builder has the same shape. It validates the concatenation with an awk BEGIN/END balance over CERTIFICATE markers only, so damage confined to a non-certificate label passes:

bundle = <one NBSP-damaged PUBLIC KEY block> + the real 132-cert bundle
builder  -> "balanced, certs=132"   -> publishes
loader   -> extras = 0              -> nothing trusted

A bundle the builder certifies as healthy, from which node loads nothing. Worth naming in "Where else it applies": the count is a model of the parser, and the parser is one spawn away.

On the rejecting direction

The PR is right that rejection is not safe here, and today's fallback is worse than the write-up suggests: a refused bundle means the launcher uses only its own CA, so a third publisher's CA is dropped for that session. Measured — 1 cert loaded, ours present, the other absent; rebuilding from the publisher files that individually load gives 2, both present. The damage lives in the merge, not in the publisher files. Only an oracle can do that repair, because only it knows which input survived.

Not asking this PR to decide

The design question is parked with Chris and me, and touches two other implementations, so this is evidence toward that decision rather than a request to change this PR's scope. The rule and both corollaries land as written regardless. My reading: the increment is 4 ms once per launch, the predicate is measurably wrong in both directions on a real bundle today, and the round count is the argument the PR itself makes.

— codeslake (CCF contributor)

@codeslake

Copy link
Copy Markdown
Contributor

Implemented the oracle in #296 and it found three more instances of this rule's own failure mode — two of them in code I wrote while applying the rule. Posting them because they sharpen where the corollaries need to bite.

The rule caught a wrong row in our own test table

Running the existing 26-shape table against the real loader, one row disagreed:

shape table said loader handshake
torn block borrowing a later END unusable 2 certs OK

That row recorded the predicate's behaviour as the expectation. Five review rounds, three parties, and it survived every one — because each round compared the code to the table and none compared the table to node. Corollary B says verify a test reaches the shipped code; this is the sibling failure: the test reached the code fine and asserted the wrong answer.

A green suite is not evidence, measured twice more

The probe API did not exist on half the runtimes we support. tls.getCACertificates lands in v22.15 / v23.10; package.json declares engines: >=18 and CI runs 18/20/22. Measured on real interpreters — v20.19.0 undefined, v22.14.0 undefined, v22.15.0 function. Every verdict on those hosts was "cannot tell", and the call site kept the bundle on "cannot tell", so the guard was not conservative — it was absent, and worse than the state before it existed. The fix is a handshake probe instead of an API probe: it works back to node 12, and it asks the contract rather than a proxy for it.

The launcher crashed on a path 43 green tests never touched. Every test built its CA with readFileSync(path, "utf8") — a string. The shipped caller uses readFileSync(path) — a Buffer. t.endsWith is not a function, swallowed by an outer catch, reported as could not evaluate ca-trust.pem — which reads as a bundle problem rather than a type error in our own code. The peer publisher's CA was dropped by the function written to keep it:

before:  extras 1   ours true   peer FALSE
after:   extras 3   ours true   peer TRUE

Neither TDD nor an intermediate-value check caught that one. What caught it was running the real entry point. That is a third corollary and I think it belongs in the section:

C. A test harness constructs its inputs; production constructs different ones. Before calling a path verified, run the shipped entry point once. Coverage of a function is not coverage of its caller.

On the cost figure

The 38 ms in "Not in scope here" is the bare-spawn floor, not the oracle. Measured, 100 interleaved runs so both share load conditions:

bare node spawn        median 16.4 ms
oracle (handshake)     median 20.4 ms
                       ─────────────
increment               median  4.0 ms

Once per launcher start, on a path that already forks node for the proxy (bin/claude-via-proxy.mjs:144).

Where else it applies — one more instance, upstream of the guard

The bundle builder has the same shape one level up. It validates the merge with an awk BEGIN/END balance over CERTIFICATE markers only, so damage confined to a non-certificate label passes:

bundle = <NBSP-damaged PUBLIC KEY block> + a healthy 132-cert bundle
builder  -> "balanced, certs=132"   -> publishes
loader   -> extras = 0              -> nothing trusted

A bundle the builder itself certifies as healthy, from which node loads nothing. And count > 0 is not enough either — openssl recovers a torn block whose body happens to be complete DER and then drops everything after it, so a merge can load exactly one certificate and have it be the wrong one. The bar that survives both is "the loader read back at least as many certificates as the inputs carry".

What the oracle does not do

Worth stating so nobody deletes the builder's check believing this replaces it: the launcher's question is self-carry. It holds one CA and can only ask about that one, so a merge that silently dropped a sibling publisher answers ok — and the dropped component falls back to its own CA and keeps working, so it is invisible to every consumer including the one that lost. Only the builder, which knows what it was handed, can see that.

Cross-version, post-implementation: full suite 1512/1512 on Linux node v24.11.1; 1511/1511 with one linux-only skip on macOS node v26.5.1 and v25.8.0.

— codeslake (CCF contributor)

Five rounds on the CA guard argued node CA-loader semantics. The client
stopped being node at CC v2.1.113 — documented in AGENTS.md, README.md,
and CHANGELOG.md, and the reason the NODE_OPTIONS preload died and this
proxy exists. Every round had it available; none consulted it.

Also adds the expectations rule: mutating the code to prove a test
reaches it is not enough when the test asserts what another program
does — the expected value must have come from that program. A shape-table
row recorded the predicate's own behaviour as the expectation and five
green rounds re-certified it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant